Electronic apparatus and learning method of electronic apparatus

ABSTRACT

Artificial intelligence for machine learning to provide an optimized response sentence in reply to an input sentence.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. § 119to Korean Patent Application No. 10-2018-0084784, filed on Jul. 20,2018, in the Korean Intellectual Property Office and Korean PatentApplication No. 10-2019-0084270, filed on Jul. 12, 2019, in the KoreanIntellectual Property Office, the disclosures of which are incorporatedby reference herein in their entireties.

BACKGROUND Field

Apparatuses and methods consistent with the disclosure relate to anelectronic apparatus and a learning method of an electronic apparatus,and more particularly, to an electronic apparatus and a learning methodof an electronic apparatus that provide a response sentence to a userbased on a reward value for an input sentence based on the inputsentence being received from the user.

Description of the Related Art

In recent years, an artificial intelligence system implementinghuman-level intelligence has been used in various fields. The artificialintelligence system is a system in which a machine performs learning anddetermination, unlike an existing rule-based smart system. As use of theartificial intelligence system is increased, a recognition rate isimproved and output based on the user input may be more accuratelyprovided, and therefore, the existing rule-based smart system has beengradually replaced by a deep learning-based artificial intelligencesystem.

Artificial intelligence technology includes machine learning (forexample, deep learning) and element technologies using the machinelearning.

Machine learning is an algorithm technology of classifying and learningfeatures of input data, and the element technology is a technology ofusing a machine learning algorithm such as deep learning, or the like,and includes technical fields such as linguistic understanding, visualunderstanding, inference/prediction, knowledge representation, a motioncontrol, and the like.

One example of technologies utilizing machine learning includes a robotsystem for use in chat and speaking, and the robot system may refer toartificial intelligence having a function to speak or interact with auser via text or voice. A conventional robot system had to use aconversation sentence tagged with a meaning or a keyword to provide aresponse sentence in response to the conversation sentence of the user.Therefore, because many conversation sentences are used in the robotsystem, tagging the meaning or the keyword with each of the conversationsentences is difficult.

SUMMARY OF THE INVENTION

Embodiments of the disclosure overcome the above disadvantages and otherdisadvantages not described above. Also, the disclosure is not requiredto overcome the disadvantages described above, and an embodiment of thedisclosure may not overcome any of the problems described above.

Aspects of the disclosure may include an electronic apparatusimplementing artificial intelligence to learn to provide a responsesentence to an input sentence of a user using a conversation sentence towhich a meaning or a keyword is not tagged, and a learning methodthereof.

According to an embodiment of the disclosure, a method of an electronicapparatus providing a response sentence in reply to an input sentence ofa user includes: obtaining a plurality of conversation sentences;obtaining a plurality of clusters for the plurality of conversationsentences based on similarity between the plurality of conversationsentences; based on the input sentence, providing the input sentence asinput to an artificial intelligence model trained to provide theresponse sentence in reply to the input sentence and providing a clusterfrom among the plurality of clusters as output from the artificialintelligence model; providing a conversation sentence from among theplurality of conversation sentences included in the cluster as theresponse sentence in reply to the input sentence of the user; obtaininga reward value for the response sentence; and updating weight values ofthe artificial intelligence model based on the reward value to optimizethe response sentence provided in reply to the input sentence.

According to another embodiment of the disclosure, an electronicapparatus includes: a memory configured to store computer-readableinstructions; and a processor configured to execute thecomputer-readable instructions control the electronic apparatus to:obtain a plurality of conversation sentences; obtain a plurality ofclusters for the plurality of conversation sentences based on similaritybetween the plurality of conversation sentences; based on an inputsentence being input from a user, provide the input sentence as input toan artificial intelligence model trained to provide a response sentencein reply to the input sentence and provide a cluster from among theplurality of clusters as output from the artificial intelligence model;provide a conversation sentence from among the plurality of conversationsentences included in the cluster as the response sentence in reply tothe input sentence of the user; obtain a reward value for the responsesentence; and update weight values of the artificial intelligence modelbased on the reward value to optimize the response sentence provided inreply to the input sentence.

According to the diverse embodiments of the disclosure as describedabove, the electronic apparatus may perform the learning for providingthe response sentence for the input sentence to the user in case thatthe input sentence is obtained from the user without the work of taggingthe meaning or the keyword for each of the plurality of conversationsentences.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The above and/or other aspects of the disclosure will be more apparentby describing certain embodiments of the present disclosure withreference to the accompanying drawings, in which:

FIG. 1 is a diagram illustrating a method by which an electronicapparatus according to an embodiment of the disclosure provides aresponse sentence to an input sentence of a user;

FIG. 2 is a block diagram illustrating a configuration of an electronicapparatus 200 according to an embodiment of the disclosure;

FIG. 3 is a diagram illustrating conversation sentences which areclustered according to an embodiment of the disclosure;

FIG. 4 is a diagram illustrating a plurality of clusters according to anembodiment of the disclosure;

FIG. 5A is a graph illustrating artificial intelligence learningexperiment data according to an embodiment of the disclosure;

FIG. 5B is a graph illustrating artificial intelligence learningexperiment data according to an embodiment of the disclosure;

FIG. 5C is a graph illustrating artificial intelligence learningexperiment data according to an embodiment of the disclosure;

FIG. 5D is a graph illustrating artificial intelligence learningexperiment data according to an embodiment of the disclosure;

FIG. 5E is a graph illustrating artificial intelligence learningexperiment data according to an embodiment of the disclosure;

FIG. 6 is a graph illustrating results obtained by performing anexperiment using all dialogue histories according to an embodiment ofthe disclosure;

FIG. 7 is a flowchart illustrating a method for providing a responsesentence according to an input sentence of a user according to anembodiment of the disclosure; and

FIG. 8 is a block diagram illustrating a configuration of an electronicapparatus according to an embodiment of the disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Hereinafter, diverse embodiments of the disclosure will be describedwith reference to the accompanying drawings.

FIG. 1 is a diagram illustrating a method by which an electronicapparatus according to an embodiment of the disclosure provides aresponse sentence in reply to an input sentence of a user.

An electronic apparatus 100 according to the disclosure may include aconversation agent 110 and an environment system (150), and theenvironment system 150 may include a dialogue history 120, a rewardgeneration module 130, and a response generation module 140.

The electronic apparatus 100 according to the disclosure may provide aresponse sentence to an input sentence of a user, compare the providedinput sentence with an existing dialogue history to determine whetherthe input sentence is appropriate, and train an artificial intelligencemodel based on a determined result. That is, the electronic apparatus100 according to the disclosure may obtain the input sentence of theuser, analyze the input sentence, and provide the response sentenceaccording to the input sentence to the user by changing an answeraccording to a conditional match of a specific sentence, word or thelike.

Specifically, the electronic apparatus 100 may obtain a plurality ofconversation sentences and obtain a plurality of clusters for theplurality of conversation sentences based on similarity of the pluralityof obtained conversation sentences. Specifically, the similarity may beobtained through a data point and a similarity metric given to each ofthe plurality of conversation sentences as described in Mathematicalexpressions 2 and 3. The conversation sentences, which are sentencesused for dialogue between users, for example, may include inputsentences of the user in a messenger application. The plurality ofclusters refer to a set of similar conversation sentences, and the abovedescription of the clusters will be described with reference to FIG. 3.

In addition, the electronic apparatus 100 may train an artificialintelligence model according to the disclosure based on the plurality ofclusters. The learning of the artificial intelligence model according tothe disclosure may be performed through a reinforcement learning method,and reinforcement learning may be performed through an interactionbetween the conversation agent 110 and the environment system 150. Thereinforcement learning is a learning method in which the artificialintelligence model selects an action according to the environment systemand performs the learning based on the environment system 150 changed bythe action. In case that the environment system 150 is changed in thereinforcement learning, the conversation agent 110 may obtain somereward and the conversation agent 110 may be learned by correcting theaction toward receiving better reward.

The environment system 150 may include the dialogue history 120, thereward generation module 130, and the response generation module 140that includes a conversation sentence to which a meaning or a keyword isnot tagged. In case that the input sentence is obtained from the user,the conversation agent 110 may select one of the plurality of clustersto provide the response sentence for the input sentence. In addition,the electronic apparatus 100 may provide one of the conversationsentences included in (belonging to) the selected cluster to the user asthe response sentence for the input sentence of the user.

In addition, the reward generation module 130 may compare the dialoguehistory 120 in the environment system 150 with the response sentenceprovided by the electronic apparatus to obtain a reward value for theresponse sentence. The reward generation module 130 may compare theconversation sentences which are pre-stored in the dialogue history 120with the response sentence to determine whether the response sentenceprovided to the user is appropriate, and may determine the reward valuefor the response sentence based on a determination result. For example,the reward generation module 130 may obtain the reward value of +1 incase that the response sentence provided to the user is appropriate fora conversation subject, and may obtain the reward value of −1 in casethat the response sentence provided to the user is out of theconversation subject.

In addition, the response generation module 140 may update the dialoguehistory 120, for example a database, by including the input sentence ofthe user and the provided response sentence in the existing dialoguehistory 120. In addition, the response generation module 140 may providethe updated dialogue history to the conversation agent 110 as new statedata.

The conversation agent 110 may update weight values of the artificialintelligence model based on the obtained reward value and the updateddialogue history. The learning of the artificial intelligence modelaccording to the disclosure may be repeatedly performed, and as thelearning is performed, a performance of the artificial intelligencemodel is improved, thereby making possible to provide an appropriateresponse sentence to the user.

FIG. 2 is a block diagram illustrating a configuration of an electronicapparatus 200 according to an embodiment of the disclosure. Theelectronic apparatus 200 according to the disclosure may be implementedas at least one server, but this is merely an example and the electronicapparatus 200 may be implemented as, for example, any computing deviceincluding a user terminal device, a smartphone, or home appliances.

The electronic apparatus 200 according to an embodiment of thedisclosure may include a memory 210 and a processor 220. The componentsillustrated in FIG. 2 are illustrative diagrams for implementing theembodiments of the disclosure, and appropriate hardware/softwareconfigurations understood by skilled in the art may be further includedin the electronic apparatus 200.

The memory 210 may store instructions or data related to one or moreother components of the electronic apparatus 200. In particular, thememory 210 may be implemented as a non-volatile memory, a volatilememory, a flash memory, a hard disk drive (HDD), a solid state drive(SDD), or the like. The memory 210 is accessed by the processor 220, andreadout, writing, correction, deletion, update, and the like of data inthe memory 210 may be performed by the processor 220. In the disclosure,a term ‘memory’ includes the memory 210, a read only memory (ROM) (notillustrated) in the processor 220, a random access memory (RAM) (notillustrated), or a memory card (not illustrated) (for example, a microsecure digital (SD) card or a memory stick) mounted in the electronicapparatus 100. In particular, the memory 210 may store a program forperforming an artificial intelligence agent. Here, the artificialintelligence agent is a personalized program for providing variousservices for the electronic apparatus 200.

In addition, the memory 210 may include a replay memory to perform thelearning of the artificial intelligence model according to an embodimentof the disclosure. The replay memory, which is a memory used for thereinforcement learning, may include a state, an action, a reward, and anext state set (s, a, r, s′). According to the disclosure, the state maybe a dialogue history, the action may be an action that the electronicapparatus 200 selects one of the plurality of clusters, and the rewardmay be a reward value for the response sentence.

The functions related to the artificial intelligence according to thedisclosure are operated through the processor 220 and the memory 210.The processor 220 may be implemented by a system-on-chip (SoC) or alarge scale integration (LSI) in which a processing algorithm isembedded, and may also be implemented in the form of a fieldprogrammable gate array (FPGA). The processor 220 may perform variousfunctions by executing computer executable instructions stored in thememory to be described later. The processor 220 may be configured as oneor a plurality of processors. Here, one or the plurality of processorsmay be a general-purpose processor such as a CPU, an AP, or the like, agraphic-dedicated processor such as a GPU, a VPU, or the like, or anartificial intelligence dedicated process such as an NPU. One or theplurality of processors performs a control to process input dataaccording to predefined operating rules or artificial intelligencemodels stored in the memory 210. The predefined operating rules orartificial intelligence models are characterized by being createdthrough learning. Here, the predefined operating rules or artificialintelligence models created through learning refer to the predefinedoperating rules or artificial intelligence models of desiredcharacteristics created by applying learning algorithms to a largenumber of learning data. The learning of the artificial intelligencemodel may be performed in a device itself in which the artificialintelligence according to the disclosure is performed, or may also beperformed through a separate server/system.

The processor 220 according to the disclosure may obtain a plurality ofconversation sentences by executing at least one command stored in thememory 210, and may obtain a plurality of clusters for the plurality ofconversation sentences based on similarity of the plurality ofconversation sentences.

In addition, the processor 220 may obtain the similarity between theplurality of conversation sentences, and may cluster the plurality ofconversation sentences based on the obtained similarity to obtain theplurality of clusters. The electronic apparatus 200 according to thedisclosure may directly generate and obtain the plurality of clusters,but is not limited thereto and may obtain the plurality of clusters fromanother server. That is, the electronic apparatus 200 may obtain theplurality of clusters from an external server by receiving the pluralityof clusters generated from the external server.

In case that the input sentence is obtained from the user, the processor220 may input the input sentence to the artificial intelligence modeltrained for providing the response sentence in reply to the inputsentence to select one of the plurality of clusters, and may provide asoutput one of the conversation sentences belonging to the selectedcluster as the response sentence in reply to the input sentence of theuser. According to an embodiment of the disclosure, the processor 220may randomly select one of the conversation sentences belonging to theselected cluster and provide the sentence to the user as the responsesentence.

In addition, the processor 220 may obtain a reward value for theprovided response sentence and update weight values of the artificialintelligence model based on the reward value. In addition, the processor220 may obtain a dialogue history, and compare the response sentencewith the conversation sentence which is pre-stored in the dialoguehistory to obtain the reward value for the response sentence.

In addition, the processor 220 may apply the updated weight values tothe artificial intelligence model, select one of the clusters based onthe artificial intelligence model to which the weight values are appliedwhen the input sentence is obtained from the user, and provide one ofthe conversation sentences belonging to the selected cluster as theresponse sentence for the input sentence of the user. That is, wheneverthe response sentence is provided to the user, the processor 220 mayupdate the dialogue history by including the input sentence of the userand the response sentence in the dialogue history. In addition, theprocessor 220 may learn the artificial intelligence model based on theupdated dialogue history and the reward value for the response sentence,and continuously update the weight values of the artificial intelligencemodel, thereby making possible to provide an appropriate responsesentence to the user.

In addition, the processor 220 may randomly select one of the pluralityof clusters with a predetermined probability, and may select a clusterhaving the maximum output value of the artificial intelligence model forthe input sentence when one of the plurality of clusters is not randomlyselected. According to an embodiment of the disclosure, thepredetermined probability may be 10%. In case that the predeterminedprobability is 10%, the processor 220 may randomly select one of theplurality of clusters with a probability of 10%, provide one of theconversation sentences belonging to the randomly selected one of theplurality of clusters as the response sentence, and select a clusterhaving the maximum output value of the artificial intelligence model forthe input sentence with a probability of 90%. According to thedisclosure, the artificial intelligence model may be a Q-function.Therefore, the processor 220 may select a cluster having the maximumQ-function value for the input sentence, and provide one of theconversation sentences belonging to the selected cluster as the responsesentence. Details of the Q-function will be described with reference toMathematical expression 4.

In addition, the processor 220 may receive the input sentence as voiceor text, and may provide the response sentence as voice or text. In casethat the input sentence is received as the text, an application to parsethe text may be utilized, and in case that the input sentence isreceived as the voice, an application that converts the voice into thetext, and the like may be utilized.

Hereinafter, the disclosure will be described with reference to FIGS. 3to 6.

In particular, the electronic apparatus 100 may obtain the plurality ofclusters based on the plurality of conversation sentences. That is, asillustrated in FIG. 3, a first cluster 310 may include conversationsentences associated with greetings (Hi. Hello. Nice to meet you.), anda second cluster 320 may include sentences associated with gratitude(Thank you very much. Thank you. Thanks.). According to the disclosure,the plurality of conversation sentences belonging to the cluster may bethe conversation sentences to which a meaning or a keyword is nottagged. That is, according to the disclosure, the cost and time of alabeling work, which is a work of tagging the meaning or the keyword tothe plurality of conversation sentences, may be reduced.

The electronic apparatus 200 according to the disclosure may performdeep reinforce learning (DRL) for learning of the artificialintelligence model, and may provide the response sentence to the userusing the plurality of clusters in which the plurality of conversationsentences are clustered through the artificial intelligence modelperforming the deep reinforce learning. The plurality of clusters may beobtained for the plurality of conversation sentences based on thesimilarity of the plurality of conversation sentences. According to anembodiment of the disclosure, the electronic apparatus 200 may obtain kclusters based on a clustering algorithm using a data point {x₁, x₂ . .. x_(n)} and a similarity metric d(x_(i), x_(i′)) given to each of theplurality of the conversation sentences, and the number of k may bedirectly set by the user.

$\begin{matrix}{x_{i} = {\frac{1}{N_{i}}{\sum\limits_{j = 1}^{N_{i}}\; c_{j}}}} & \left\lbrack {{Mathematical}\mspace{14mu} {expression}\mspace{14mu} 1} \right\rbrack\end{matrix}$

According to the disclosure, each data point x_(i) may be generated foreach of the conversation sentences using mean word embedding for eachconversation sentence through Mathematical expression 1. c_(j) is acoefficient vector of j and N_(i) shows the number of words of asentence i.

d(x _(l) ,x _(l′))=√{square root over (Σ_(j=1) ^(m)(x _(l) ^(j) −x _(l′)^(j))²)}  [Mathematical expression 2]

According to the disclosure, a similarity metric d(x^(j) _(i), x^(j)_(i)) may be generated for m words through Mathematical expression 2.

According to the disclosure, the clusters for the plurality ofconversation sentences may be obtained through Mathematical expressions1 and 2 described above.

In addition, the electronic apparatus 200 may update weight values ofthe artificial intelligence model based on the reward value for theprovided response sentence. Here, an interactive reward value (function)R_(i) according to the disclosure for updating the weight values is asshown in Mathematical expression 3.

R _(i)=Σ_(j=1) ^(N) r _(j) ^(i)(a)  [Mathematical expression 3]

i shows a dialogue between the user and the electronic apparatus, and jrelates to the response sentence provided by the electronic apparatus.r_(j) ^(i)(a) may show +1 in case that the response sentence provided bythe electronic apparatus is appropriate for a conversation subject, andmay show −1 in case that the response sentence provided by theelectronic apparatus is out of the conversation subject.

${r_{j}^{i}(a)} = \begin{pmatrix}{{+ 1},{{if}\mspace{14mu} a\mspace{14mu} {is}\mspace{14mu} a\mspace{14mu} {human}\mspace{14mu} {response}\mspace{14mu} {in}\mspace{14mu} {dialogue}\text{-}{turn}\mspace{14mu} i},j} \\{{- 1},{{if}\mspace{14mu} a\mspace{14mu} {is}\mspace{14mu} {human}\mspace{14mu} {but}\mspace{14mu} {randomly}\mspace{14mu} {chosen}\mspace{14mu} ({incoherent})}}\end{pmatrix}$

TABLE 1 Sentences with reward r = +1 Sentences with reward r = −1 hellowhat are doing today? hello what are doing today? i'm good, i just gotoff work and tired, i do your cats like candy? have two jobs. i just gotdone watching a horror movie i just got done watching a horror i ratherread, i have read about 20 books movie this year. do you have anyhobbies? wow! i do love a good horror movie. wow! i do love a goodhorror loving this cooler weather movie, loving this cooler weather buta good movie is always good. good job ! if you live to 100 like me, youwill need all that learning. yes! my son is in junior high and i justyes! my son is in junior high and started letting him watch them i juststarted letting him watch i work in the movies as well. them what a nicegesture. i take my dog to compete in agility classes.

Table 1 is a table for determining a reward value according to theresponse sentence according to an embodiment of the disclosure.

As shown in Table 1, in case that the response sentence provided by theelectronic apparatus 200 is appropriate for the input sentence of theuser, the reward value of +1 may be obtained, but in case that theresponse sentence is inappropriate for the input sentence of the user,the reward value of −1 may be obtained.

In addition, the electronic apparatus 200 may perform the learningthrough a process of maximizing a Q-function (Q*) showing an accumulatedreward value. That is, the electronic apparatus 200 may select a clusterhaving the maximum Q-function value for the input sentence of the useramong the plurality of clusters, and provide one of the conversationsentences belonging to the selected cluster to the user as the responsesentence.

The accumulated reward value (Q*) may be as shown in Mathematicalexpression 4.

Q*(s,a;θ)=max_(π) E[r _(t) +γr _(t+1) +γr _(t+2) + . . . Cs,a,π_(θ)]  [Mathematical expression 4]

In Mathematical expression 4, r shows a reward as a result of selectingan action a in a state sofa time interval t, γ is a discounting factor,and Q*=(s,a;θ), which is an optimal action-value function, is theQ-function. The electronic apparatus 200 may explore a new (state,action) pair in order to improve the reward value, or select an actionin a probabilistic manner to utilize an already learned value. During atest process, the electronic apparatus 200 may select the actionaccording to a*=argmax_(a|-|A)Q*(s,a;θ).

A learning performing process of the electronic apparatus according tothe disclosure will be described with reference to an algorithm 1.

The electronic apparatus 200 may perform reinforcement learning of theartificial intelligence model using generalization of a DQN method [21].

Algorithm 1 Chat DQN Learning  1: Initialise Deep Q-Networks with replaymemory D, dialogue history H, action-value function Q with randomweights Θ, and target action-value functions {circumflex over (Q)} with{circumflex over (Θ)} = Θ  2: Initialise clustering model from trainingdialogue data  3: repeat  4:  Sample a training dialogue (human-humansentences)  5:  Append first sentence to dialogue history H  6:  S =word embedding representation of H  7:  repeat  8:   Generate noisycandidate response sentences  9:   A = cluster IDs of candidate responsesentences 10:    $a = \left\{ \begin{matrix}{rand}_{a \in A} & {{{if}\mspace{14mu} {random}\mspace{14mu} {number}} \leq ɛ} \\\max_{a \in A} & {{Q\left( {s^{\prime},{a^{\prime};\hat{\Theta}}} \right)}\mspace{14mu} {otherwise}}\end{matrix} \right.$ 11:   Execute chosen clustered action a 12:  Observe human-likeness dialogue reward r 13:   Observe environmentresponse (agent's partner) 14:   Append agent and environment responsesto H 15:   s = word embedding representation of H 16:   Appendtransition (s, a, r, s′) to D 17:   Sample random minibatch (s_(j),a_(j), r_(j), s′_(j)) from D 18:    $y_{j} = \left\{ \begin{matrix}{r_{j}\mspace{14mu} {if}\mspace{14mu} {final}\mspace{14mu} {step}\mspace{14mu} {of}\mspace{14mu} {episode}} \\{r_{j} + {\gamma \; {\max_{a \in A}{{\hat{Q}\left( {s^{\prime},{a^{\prime};\hat{\Theta}}} \right)}\mspace{14mu} {otherwise}}}}}\end{matrix} \right.$ 19:   Set err = (y_(j) − Q(s′, a′; Θ))² 20:  Gradient descent step on err with respect to Θ 21:   Reset {circumflexover (Q)} = Q every C steps 22:   s ← s′ 23:  until s is end of dialogue24: until convergence

Referring to algorithm 1, the electronic apparatus 200 may initialize areplay memory D, a dialogue history H, and a Q-function (Q, {circumflexover (Q)}), and may sample a learning dialogue from the dialogue historybetween the humans. When the user enters an input sentence, the dialogueis started, and the electronic apparatus 200 may generate a plurality ofclusters including a response sentence for an actual response of a humanand a randomly selected response sentence. The plurality of clusters mayinclude the plurality of clustered conversation sentences as describedabove, and the electronic apparatus 200 may obtain an action ofselecting one of the plurality of clusters based on the plurality ofclusters. The electronic apparatus 200 may randomly select one of theplurality of clusters with a predetermined probability (∈), and mayselect a cluster having the maximum output value of the artificialintelligence model for the input sentence when it does not randomlyselect one of the plurality of clusters. The predetermined probabilitymay be 10% as described above. The obtained action may be conveyed tothe environment system S, and the reward (value) may be determined basedon the obtained action. In addition, the electronic apparatus 200 mayupdate the dialogue history H by including the response sentence for theinput sentence of the user in the dialogue history H. Through theprocess described above, a set (s, a, r, s′) of a state s, an action a,a reward r, a next state s′ may be updated in the replay memory D, and anew word embedding may be generated from the dialogue history H by theupdated replay memory D. In addition, in order to update weights Θ ofthe artificial intelligence model, experiment data MB=(s_(j), a_(j),r_(j), s′_(j)) by a minibatch method may be sampled from the replaymemory D. The updating of the weights Θ of the artificial intelligencemodel may be performed according to Mathematical expression 5.

L(θ_(j))=

_(MB)[r+γmax_(a′) {circumflex over (Q)}(s,a;θ _(j))−Q(s,a;{circumflexover (θ)} _(j))][Mathematical expression 5]

The Q-function {circumflex over (Q)}, which is the target function, andthe state s may be updated according to the updating of the weights, andthe learning of the artificial intelligence model may be performed bycontinuously performing the process of updating the weights describedabove.

Hereinafter, a process of performing an experiment with respect to alearning method of an electronic apparatus according to the disclosurewill be described with reference to FIGS. 4 to 6.

In the experiment of the learning method according to the disclosure,the experiment was performed based on 100 clusters as illustrated inFIG. 4. As the number of clusters increases, the problem of providingthe same cluster to the user may be solved, but as the number ofclusters increases, the calculation cost may also increase. Therefore,in the experiment according to the disclosure, the experiment wasperformed using 100 clusters, and in the 100 clusters, conversationsentences representing greetings may be assigned to the same cluster andquestion sentences representing preferences may be assigned to the samecluster.

FIGS. 5A to 5E are graphs illustrating artificial intelligence learningexperiment data according to an embodiment of the disclosure.

In the experiment according to the disclosure, the experiment wasperformed using 100 learning dialogue histories, and the experiment wasalso performed using all learning dialogue histories. In the experimentusing the 100 learning dialogue histories, 20,000 learnings wereperformed, and in the experiment using all leaning dialogue histories,120,000 learnings were performed. In addition, in each experiment, theexperiment was performed based on 100 clusters. Referring to FIGS. 5A to5E, even in case that the electronic apparatus 200 performs an action ofproviding 100 response sentences with 100 experiments, performance isimproved. In the graph, each point may represent one dialogue. That is,each point may represent one response sentence corresponding to theinput sentence of the user.

FIG. 5A is a graph illustrating results of an experiment based on 0 to100 dialogue histories, FIG. 5B is a graph illustrating results of anexperiment based on 100 to 200 dialogue histories, FIG. 5C is a graphillustrating results of an experiment based on 200 to 300 dialoguehistories, FIG. 5D is a graph illustrating results of an experimentbased on 300 to 400 dialogue histories, and FIG. 5E is a graphillustrating results of an experiment based on 400 to 500 dialoguehistories.

In each FIG., the top graph (1) illustrates a result obtained byperforming 20,000 learnings, and the bottom graph (2) illustrates aresult obtained by performing 120,000 learnings. Referring to eachgraph, as the learning is performed, the accumulated reward value tendsto increase.

FIG. 6 is a graph illustrating results obtained by performing anexperiment using all dialogue histories according to an embodiment ofthe disclosure.

The graph of FIG. 6 illustrates the result obtained by performing theexperiment using all dialogue histories, unlike FIGS. 5A to 5E. Comparedwith the experiment using some dialogue histories, it may be seen thatthe performance of artificial intelligence models is better when theexperiment is performed using all dialogue histories.

TABLE 2 Data |Embedding| = 100 |Embedding| = 300 Splits Training TestTraining Test  0-100 2.5049 −2.0305 2.4145 −0.9488 100-200 2.3676−2.1609 2.4602 −0.5761 200-300 1.9599 −2.0796 2.4883 −0.8967 300-4001.9288 −2.2653 2.5492 −0.9488 400-500 2.2941 −2.1830 2.2389 −1.0070500-600 2.5941 −2.0440 2.5146 −2.2715   0-17877 −0.4027 −1.8757 −0.2885−0.4539 Average 1.8924 −2.0913 2.0539 −1.0147 Upper Bound TBF 7.5942 TBF7.5942 Lower Bound TBF −7.7276 TBF −7.7276 Random −2.4139 −2.5526−2.4139 −2.5526

Table 2 is a table illustrating the results obtained by performing20,000 learnings using 100 and 300 clusters.

Referring to Table 2, the result obtained by performing 20,000 learningsusing the 100 clusters (|Embedding|=100), and the result obtained byperforming 20,000 learnings using the 300 clusters (|Embedding|=300) areillustrated. Referring to Table 2, performing of the learning using the300 clusters shows higher performance. In addition, as the dialoguehistories (Data Splits) are used, higher performance is obtained.

TABLE 3 Data |Embedding| = 100 |Embedding| = 300 Splits Training TestTraining Test  0-100 0.7349 TBF 1.0104 −2.3377 100-200 1.1612 TBF 1.7355−1.6193 200-300 0.8504 TBF 2.3953 −1.3901 300-400 2.4840 TBF 2.5090−1.5039 400-500 0.8349 TBF 2.8329 −1.6111 500-600 0.7853 TBF 2.6007−1.5689   0-17877 −0.3060 TBF −0.2931 −1.2652 Average 0.9350 TBF 1.8272−1.6137

Table 3 is a table illustrating the results obtained by performing120,000 learnings using 100 and 300 clusters.

Referring to Table 3, the result obtained by performing 120,000learnings using the 100 clusters (|Embedding|=100), and the resultobtained by performing 120,000 learnings using the 300 clusters(|Embedding|=300) are illustrated.

FIG. 7 is a flowchart for describing a method for providing a responsesentence according to an input sentence of a user according to anembodiment of the disclosure.

The electronic apparatus 200 may obtain a plurality of conversationsentences (S710). The plurality of conversation sentences may besentences such as ‘Hi’ and ‘Thank you very much’ as described above withreference to FIG. 3. In addition, the electronic apparatus 200 mayobtain a plurality of clusters for the plurality of conversationsentences based on similarity between the plurality of conversationsentences (S720). That is, the electronic apparatus 200 may obtain thesimilarity between the plurality of conversation sentences, and maycluster the plurality of conversation sentences based on the obtainedsimilarity to obtain the plurality of clusters. In addition, if theelectronic apparatus 200 obtains an input sentence from a user, theelectronic apparatus 200 may select one of the plurality of clusters(S730). According to an embodiment of the disclosure, the electronicapparatus 200 may randomly select one of the plurality of clusters witha predetermined probability. In case that the electronic apparatus 200does not randomly select one of the plurality of clusters, theelectronic apparatus 200 may select a cluster having the maximum outputvalue of an artificial intelligence model for the input sentence. Inaddition, the electronic apparatus 200 may provide one of theconversation sentences belonging to the selected cluster as a responsesentence (S740). According to an embodiment of the disclosure, theelectronic apparatus 200 may randomly select one of the conversationsentences belonging to the selected cluster and provide it as theresponse sentence. In addition, the electronic apparatus 200 may obtaina reward value for the provided response sentence (S750). According toan embodiment of the disclosure, the reward value may be obtainedthrough Mathematical expression 3. In addition, the electronic apparatus200 may update weights of the artificial intelligence model based on theobtained reward value (S760). That is, the electronic apparatus 200 mayupdate the weights by learning the artificial intelligence model in adirection in which the reward value for the response sentence for theinput sentence of the user is large.

FIG. 8 is a block diagram illustrating a specific configuration of anelectronic apparatus according to an embodiment of the disclosure. Anelectronic apparatus 800 according to the disclosure may be implementedas an electronic apparatus such as a smartphone, and the electronicapparatus 800 may include a memory 810, a processor 820, a communicator830, a display 840, and a touch sensor 850.

A detailed description for the components overlapped with componentsillustrated in FIG. 2 among the components illustrated in FIG. 8 will beomitted.

The communicator 830 is a component performing communication withvarious types of external devices in various types of communicationmanners. The communicator 830 may include a wireless fidelity (Wi-Fi)chip, a Bluetooth chip, a wireless communication chip, and a near fieldcommunication (NFC) chip. The processor 820 may perform communicationwith various external devices using the communicator 830. In particular,the Wi-Fi chip and the Bluetooth chip perform communication in a Wi-Fimanner and a Bluetooth manner, respectively. In case of using the Wi-Fichip or the Bluetooth chip, various kinds of connection information suchas a service set identifier (SSID), a session key, and the like, arefirst transmitted and received, communication is connected using theconnection information, and various kinds of information may then betransmitted and received. The wireless communication chip means a chipperforming communication depending on various communication standardssuch as Institute of Electrical and Electronics Engineers (IEEE),Zigbee, 3^(rd) generation (3G), 3^(rd) generation partnership project(3GPP), long term evolution (LTE), and the like. The NFC chip means achip operating in the NFC manner using a band of 13.56 MHz among variousradio frequency identification (RFID) frequency bands such as 135 kHz,13.56 MHz, 433 MHz, 860 to 960 MHz, 2.45 GHz, and the like.

The communicator 830 may perform communication with an external deviceand a server, and according to an embodiment of the disclosure, thecommunicator 830 may obtain a plurality of conversation sentences or aplurality of clusters from the external device or the server. That is,the electronic apparatus 800 may directly generate the plurality ofconversation sentences and the plurality of clusters, but is not limitedthereto, and the electronic apparatus 800 may obtain the plurality ofconversation sentences or the plurality of clusters from the externaldevice or the server through the communicator 830.

The display 840 may display a response sentence provided by theprocessor 820 to a user. In addition, according to an embodiment of thedisclosure, when the response sentence is displayed on the display 840,the user directly evaluates whether the response sentence is appropriatesuch that a reward value for the response sentence may be obtained. Theevaluation for the response sentence of the user may be performed byseparately displaying a user interface (UI) on the display 840. Thedisplay 840 may be implemented as a liquid crystal display (LCD), anorganic light emitting display (OLED), a plasma display panel (PDP), orthe like, but is not limited thereto.

The touch sensor 850 may receive an input sentence of the user. That is,the touch sensor 850 may sense a touch input of the user on the display840. The touch of the user may include a touch input by a finger of theuser and a touch input by an electronic pen. In case of the touch inputby the finger of the user, the touch sensor 850 may sense the touchinput and output an input sentence corresponding to a sensed touchsignal. Here, information corresponding to the touch signal may bedisplayed on the display 840.

Although not illustrated in FIG. 8, in case that the input sentence ofthe user is input through voice, the electronic apparatus 800 mayfurther include a microphone or the like.

Meanwhile, it is to be understood that the descriptions described aboveare not limited to specific embodiments, but include variousmodifications, equivalents, and/or alternatives according to embodimentsof the disclosure. Throughout the accompanying drawings, similarcomponents will be denoted by similar reference numerals.

In the disclosure, a term “user” may be a person that uses theelectronic apparatus or an apparatus (e.g., an artificial intelligenceelectronic apparatus) that uses the electronic apparatus.

The artificial intelligence model according to the disclosure mayinclude a plurality of neural network layers. Each layer has a pluralityof weight values and a layer calculation is performed by calculating acalculation result of a previous layer and the plurality of weightvalues. Examples of the neural network include a convolutional neuralnetwork (CNN), a deep neural network (DNN), a recurrent neural network(RNN), a restricted boltzmann machine (RBM), a deep belief network(DBN), a bidirectional recurrent deep neural network (BRDNN), and deepQ-networks, and the neural network in the disclosure is not limited tothe examples described above except for the case in which it isspecified.

A learning algorithm is a method of training a predetermined targetdevice (e.g., a robot) using a plurality of learning data so that thepredetermined target device may make a decision or predict itself.Examples of the learning algorithm include supervised learning,unsupervised learning, semi-supervised learning, or reinforcementlearning, and the learning algorithm in the disclosure is not limited tothe examples described above except for the case in which it isspecified.

The electronic apparatus according to the disclosure may perform thelearning of the artificial intelligence model through reinforcementlearning, unsupervised learning, and deep learning to provide theresponse sentence for the input sentence of the user.

Meanwhile, according to an embodiment of the disclosure, the diverseembodiments described hereinabove may be implemented by softwareincluding instructions that are stored in machine (e.g., acomputer)-readable storage media. The machine is an apparatus thatinvokes the stored instructions from the storage media and is operableaccording to the invoked instructions, and may include the electronicapparatus (e.g., an electronic apparatus A) according to the disclosedembodiments. When the instructions are executed by the processor, theprocessor may perform functions corresponding to the instructions,either directly or using other components under the control of theprocessor. The instructions may include codes generated or executed by acompiler or an interpreter. The machine-readable storage media may beprovided in the form of non-transitory storage media. Here, the terms‘non-transitory’ means that the storage media do not include a signaland is tangible, but do not distinguish whether data is storedsemi-permanently or temporarily in the storage media.

In addition, according to an embodiment of the disclosure, the methodaccording to the diverse embodiments described above may be included andprovided in a computer program product. The computer program product maybe traded as a product between a seller and a purchaser. The computerprogram product may be distributed in the form of a machine readablestorage media (e.g., a compact disc read only memory (CD-ROM)), oronline through an application store (e.g., PlayStore™). In case of theonline distribution, at least a portion of the computer program productmay be at least temporarily stored in a storage medium such as a memoryof a server of a manufacturer, a server of an application store, or arelay server, or be temporarily generated.

Each of the components (e.g., modules or programs) according to thediverse embodiments described above may include a single entity or aplurality of entities, and some sub-components of the sub-componentsdescribed above may be omitted, or other sub-components may be furtherincluded in the diverse embodiments. Alternatively or additionally, somecomponents (e.g., modules or programs) may be integrated into one entityto perform the same or similar functions performed by the respectivecomponents prior to the integration. The operations performed by themodule, the program, or other component, in accordance with the diverseembodiments may be performed in a sequential, parallel, iterative, orheuristic manner, or at least some operations may be executed in adifferent order or omitted, or other operations may be added.

Although the embodiments of the disclosure have been illustrated anddescribed hereinabove, the disclosure is not limited to theabovementioned specific embodiments, but may be variously modified bythose skilled in the art to which the disclosure pertains withoutdeparting from the gist of the disclosure as disclosed in theaccompanying claims. These modifications should also be understood tofall within the scope and spirit of the disclosure.

What is claimed is:
 1. A method of an electronic apparatus providing aresponse sentence in reply to an input sentence of a user, the methodcomprising: obtaining a plurality of conversation sentences; obtaining aplurality of clusters for the plurality of conversation sentences basedon similarity between the plurality of conversation sentences; based onthe input sentence, providing the input sentence as input to anartificial intelligence model trained to provide the response sentencein reply to the input sentence and providing a cluster from among theplurality of clusters as output from the artificial intelligence model;providing a conversation sentence from among the plurality ofconversation sentences included in the cluster as the response sentencein reply to the input sentence of the user; obtaining a reward value forthe response sentence; and updating weight values of the artificialintelligence model based on the reward value to optimize the responsesentence provided in reply to the input sentence.
 2. The method asclaimed in claim 1, wherein a meaning or a keyword is not tagged to eachof the plurality of conversation sentences.
 3. The method as claimed inclaim 1, wherein obtaining the plurality of clusters comprises:obtaining the similarity between the plurality of conversationsentences; and clustering the plurality of conversation sentences basedon the similarity between the plurality of conversation sentences toobtain the plurality of clusters.
 4. The method as claimed in claim 1,further comprising obtaining a dialogue history, wherein obtaining thereward value comprises obtaining the reward value by comparing theresponse sentence with the dialogue history.
 5. The method as claimed inclaim 4, further comprising storing the input sentence and the responsesentence in the dialogue history.
 6. The method as claimed in claim 1,wherein updating the weight values comprises obtaining the reward valuein response to providing the response sentence; and wherein the updatingcomprises updating the weight values to maximize an accumulated value ofthe reward value.
 7. The method as claimed in claim 1, furthercomprising: applying the updated weight values to the artificialintelligence model; and selecting an updated cluster from among theplurality of clusters in response to the input sentence input to theartificial intelligence model to which the updated weight values areapplied; and providing an updated conversation sentence from among theplurality of conversation sentences included in the updated cluster asthe response sentence in reply to the input sentence of the user.
 8. Themethod as claimed in claim 7, wherein selecting the cluster from amongthe plurality of clusters comprises: randomly selecting the cluster fromamong the plurality of clusters based on a predetermined probability, orselecting the cluster based on a maximum output value of the artificialintelligence model in reply to the input sentence.
 9. The method asclaimed in claim 1, wherein the input sentence is voice or text, andwherein the providing comprises providing the response sentence as thevoice or the text.
 10. An electronic apparatus comprising: a memoryconfigured to store computer-readable instructions; and a processorconfigured to execute the computer-readable instructions control theelectronic apparatus to: obtain a plurality of conversation sentences;obtain a plurality of clusters for the plurality of conversationsentences based on similarity between the plurality of conversationsentences; based on an input sentence being input from a user, providethe input sentence as input to an artificial intelligence model trainedto provide a response sentence in reply to the input sentence andprovide a cluster from among the plurality of clusters as output fromthe artificial intelligence model; provide a conversation sentence fromamong the plurality of conversation sentences included in the cluster asthe response sentence in reply to the input sentence of the user; obtaina reward value for the response sentence; and update weight values ofthe artificial intelligence model based on the reward value to optimizethe response sentence provided in reply to the input sentence.
 11. Theelectronic apparatus as claimed in claim 10, wherein a meaning or akeyword is not tagged to each of the plurality of conversationsentences.
 12. The electronic apparatus as claimed in claim 10, whereinthe processor is configured to: obtain the similarity between theplurality of conversation sentences, and cluster the plurality ofconversation sentences based on the similarity between the plurality ofconversation sentences to obtain the plurality of clusters.
 13. Theelectronic apparatus as claimed in claim 10, wherein the processor isconfigured to: obtain a dialogue history, and obtain the reward valuefor the response sentence by comparing the response sentence with thedialogue history.
 14. The electronic apparatus as claimed in claim 13,wherein the processor is configured to store the input sentence and theresponse sentence in the dialogue history.
 15. The electronic apparatusas claimed in claim 10, wherein the processor is configured to: obtainthe reward value in response to providing the response sentence, andupdate the weight values to maximize an accumulated value of the rewardvalue.
 16. The electronic apparatus as claimed in claim 10, wherein theprocessor is configured to: apply the updated weight values to theartificial intelligence model, select an updated cluster from among theplurality of clusters in response to the input sentence input to theartificial intelligence model to which the updated weight values areapplied, and provide an updated conversation sentence from among theplurality of conversation sentences included in the updated cluster asthe response sentence in reply to the input sentence of the user. 17.The electronic apparatus as claimed in claim 16, wherein the processoris configured to: randomly select the cluster from among the pluralityof clusters based on a predetermined probability, or select the clusterbased on a maximum output value of the artificial intelligence model inreply to the input sentence.
 18. The electronic apparatus as claimed inclaim 10, wherein the processor is configured to: receive the inputsentence as voice or text, and provide the response sentence as thevoice or the text.